Skip to content

Add server health monitoring to detect and auto-restart when interval server loses connection#2

Draft
codegen-sh[bot] wants to merge 1 commit into
mainfrom
sw-49-interval-server-monitor
Draft

Add server health monitoring to detect and auto-restart when interval server loses connection#2
codegen-sh[bot] wants to merge 1 commit into
mainfrom
sw-49-interval-server-monitor

Conversation

@codegen-sh
Copy link
Copy Markdown

@codegen-sh codegen-sh Bot commented Aug 15, 2025

Overview

This PR adds a comprehensive server health monitoring system to detect and automatically recover from the "Nothing here yet" issue that occurs when the interval server loses connection to hosts.

Key Changes

  1. New Server Health Monitoring System:

    • Added serverHealth.ts with health checks that monitor the server's state
    • Implements auto-restart capability when the server is detected to be in an unhealthy state
    • Uses a cooldown period to prevent restart loops

  2. Enhanced Connection Monitoring:

    • Improved logging for connection creation and closure
    • Added detailed tracking of ping failures
    • Better diagnostics for connection issues

  3. Automatic Recovery:

    • Server will automatically restart itself when it detects a state that would cause the "Nothing here yet" issue
    • Properly cleans up database state before restarting to ensure clean reconnection

How It Works

The system performs periodic health checks that:

  1. Verify database connectivity
  2. Compare connected hosts with database records
  3. Check for inconsistencies between WebSocket connections and database state
  4. Automatically restart the server when multiple consecutive checks fail

This should eliminate the need for manual restarts when the server gets into the state where it shows "Nothing here yet" in the web UI.

Testing

This can be deployed to Railway and will automatically monitor itself. The logs will show detailed information about the server's health state and any actions taken to recover from issues.

Fixes SW-49

💻 View my work • 👤 Initiated by @zeevl • About Codegen
⛔ Remove Codegen from PR • 🚫 Ban action checks

@codegen-sh @zeevl
Add server health monitoring to detect and auto-restart when interval…
d8fd8e6 
… server loses connection

- Add serverHealth.ts with comprehensive health checks
- Enhance connection monitoring with better logging
- Add auto-restart capability when server is in unhealthy state
- Track ping failures and connection statistics

Co-authored-by: Steve Lamb <steve@getscribeware.com>
@linear
Copy link
Copy Markdown

linear Bot commented Aug 15, 2025

SW-49 interval server keeps losing connection?

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Aug 15, 2025

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Join our Discord community for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@zeevl zeevl

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zeevl